ABSTRACT
We investigated causes of interrater disagreements in the observational assessment of clinical training for first-year postgraduate trainees. In 1998 25 first-year postgraduates rotated through the Second Department of Surgery for 3 months, including 1 month in cardiovascular surgery, thoracic surgery, and upper-gastrointestinal surgery. Each trainee cared for several patients at most with a senior resident under the supervision of senior staff members. Nine attending physicians (staff members), 3 doctor-course graduates, and 2 chief residents assessed the trainees at the end of the rotation with special reference to clinical, social, and supervisory abilities. Trainees were given scores of “Good, ” “Fair, ” “Pass, ” or “Fail” for each ability. Interrater disagreements often involved responsibility and activeness, which reflected social abilities, and rapid patient consultations, orderly arrangement of laboratory examinations and procedures, and avoiding ordering of unnecessary laboratory examinations and medications, which reflected supervisory abilities. Assessments of poorly performing trainees often disagreed. Some interrater disagreements were seen among 4 of 14 attending physicians, but disagreements were fewer among the 3 doctor-course graduates and 2 chief residents who were graduates of the college. Both the proper training of assessors and a good relationship between assessors and rotators are necessary to make appropriate evaluations that might affect the career of postgraduates trainees.